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Using Developmental Research to Study One’s Teaching of an Instructional Design Course 



Neal Shambaugh 
West Virginia University 
Susan G. Magliaro 
Virginia Tech 



Abstract 

This five-year study of two instructors teaching a master's level instructional design course used 
developmental research to systematically examine a reflexive teaching approach. The reflexive teaching 
model is described. Eight data sources across six deliveries of the course were analyzed in terms of design 
decisions (i.e., the teaching model), model implementation, and model evaluation. Methodological issues of 
this developmental research study are discussed, including data management of evolving data sources, 
data analysis of teaching artifacts, and the balancing of teaching goals and research objectives. Guidelines 
for using developmental research to study one's teaching are summarized. 

Developmental Research 

We view developmental research as co-contributing to the development of educational interventions 
(models and other processes, courses, media/technology artifacts), as well as knowledge about this development in 
the form of design principles or frameworks. Developmental research uses a developmental cycle, consisting of 
design, development, implementation, and evaluation activities, to formally study these interventions over time. The 
intent of developmental research is partly pragmatic design, developing products or processes that are needed to 
serve human needs (Norman, 1993). The intent is also to learn from our design efforts, to formulate what we learn 
so that these principles and frameworks assist us in developing other interventions in different contexts. 

Reeves (2000) reminds us that the aims of any research depend on the researcher's epistemological lens, 
which in turn influences the selection of one's research goals and research framework. For example, development 
goals differ from empirical goals. While empirical research is characterized by a researcher using research 
hypotheses and the testing of these hypotheses to refine existing theory, developmental research involves 
collaborators in the analysis of practical problems with the testing of designed solution in actual practice. The 
outcome of developmental research is initially a greater understanding of the educational artifact, process, or 
intervention and ideally design principles generalizable at some level, whether it be to refine the artifact, process, or 
intervention under study or transfer to other applications or contexts. 

Richey and Nelson (1996) provide a comprehensive review of developmental research projects, which they 
classified as either Type 1 or Type 11 studies. Type 1 studies involve the design and development of products or 
processes within a developmental cycle, while in Type 11 studies researchers study what has already been developed 
with the intent of abstracting design principles for re-use. Meanwhile, van den Akker (1999) labels Type I as 
formative studies in which the goal is optimizing an intervention, while Type II studies are labeled as reconstructive 
studies and formulate design principles. In addition, van den Akker characterizes design principles to include 
substantive knowledge on "What to design?" and procedural knowledge, or "How should it be developed?" To add 
to our knowledge of what to design and how to design, says van den Akker, design principles should be justified by 
theoretical arguments, procedural detail, empirical evidence, and validated in multiple contexts. 

Informing developmental research is formative research (e.g., Newman, 1990), which (a) identifies 
shortcomings and providing suggestions to a product or a process under development; (b) uses triangulation of 
methods, instruments, sources, and settings; and (c) evaluates quality criteria, such as effectiveness 
appeal/practicality, and efficiency. Complementing formative research are design experiments (Brown, 1992), 
involving the design and study of learning environments addressing complex learning problems in actual learning 
settings with practitioners. In using design experiments Brown reminds us how mutually informative studies done 
in classroom and laboratory settings can be. Decisions facing the researcher in design experiments, according to 
Brown, include participant size, involving individualized cases studying subject traits or the use of many subjects 
looking at a single variable. A second decision deals with changes over a chosen length of time, including cross- 
sectional studies from different groups, longitudinal data from a group over time, or microgenetic studies examining 
data over a short period of time. A third decision is the appropriate choice or mix of quantitative and qualitative 
techniques to use. 
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This paper discusses the methodological issues of using a developmental research framework to 
systematically design, implement, and evaluate a process; in this case, a reflexive approach for teaching instructional 
design (ID). The next section of the paper describes the developmental study, including the reflexive teaching 
model, the instructional sequence of the ID course, the developmental research framework, and study conclusions. A 
subsequent section will discuss methodological issues of developmental research, including data management and 
analysis, and the challenges of teaching and studying one’s teaching. A final section provides guidelines on teaching 
decisions and matching developmental research prompts. 

Developmental Study of ID Instruction 

Reflexive Teaching Model 

The major components of our reflexive approach for teaching instructional design include (a) 
characteristics and roles of instructor and learner, (b) co-participation structures, and (c) dialogue of teacher and 
student within each participation structure. Teachers and students are viewed as learners, each possessing unique 
learning beliefs, knowledge, competencies, experiences, sensibilities, and motivations. The instructor roles within 
the model include that of a learner, a designer of an instructional environment, and a teacher responsive to learner 
needs within this environment. The teacher assumes a supportive role, not unlike that of a coach (Schon, 1987). In 
terms of knowledge and competencies, the teacher must bring not only expertise in instructional design, but subject 
matter knowledge, pedagogical knowledge, pedagogical content knowledge, and knowledge of the learning 
principles of instructional design (Shulman, 1986). Student roles in the reflexive model include that of a learner with 
a willingness to engage within the participation structures and perform learning tasks. By reflexive we mean 
instructor and student learning of instructional design through multiple forms of activity. 

Participation structures include the classroom, learning tasks (outside of the classroom), individual 
conferences, electronic mail, web site, and texts. Although careful consideration must be given to the design of these 
structures, some negotiation of their features by students is also encouraged. The key is being open to feedback and 
periodically “stepping outside” a teacher's perspective to consider these suggestions. Dialogue between the 
participants is crucial within these structures. For example, in a group activity, dialogue enables the knowledge of 
instructional design and one’s views and experiences to be shared in an open and testable way, initiating a shared 
reflective process. Cooperative learning, presentations, and peer/teacher evaluations are key strategies. Within the 
structures dialogue with oneself through reflective tasks promotes ID understanding and understanding of one’s own 
thought processes. Weekly written project drafts and feedback on one’s performance help to develop reflective 
activity. 

Instructional Sequence 

The first two phases in the ID process in this course, Learning Beliefs and Design Tools (principally ID 
models), are used to establish the context for the traditional ID process (see Figure 1). Design-A-Lesson and 
Learning Principles tasks help students to reflect on how they currently plan instruction and their views on learning. 
Students draft a Mission Statement, which is used to assess how students' learning beliefs are being applied in their 
projects. Students also sketch a visual of their own ID model and provide an explanatory narrative. 
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Fi gure 1 . Course sequence and learning tasks. 

Students choose an instructional problem for an ID project and record their initial understanding of the 
problem through an Intent Statement. The Needs Assessment phase structures students' research into the 
instructional problem and options for addressing it. A personal conference provides individual assistance with their 
needs assessment strategy in terms of what to study, with whom to talk, what references to consult, and how to 
summarize their findings. Based on Needs Assessment, students identify goals for their project. Following Needs 
Assessment, students are introduced to design phases, which include Instructional Sequence, Assessment, 
Instructional Framework, Instructional Media /Technology, and a Prototype Lesson. Instruction helps raise students* 
awareness to the purposes of assessment and appropriate assessment methods. During the Instructional Frameworks 
phase in which teaching options are specified, students demonstrate a teaching strategy in their Prototype Lesson. 
Instructional Media and Technology is addressed throughout the course beginning with Needs Assessment. We urge 
students to be open to a range of media and technology possibilities and to make a case for how their choices 
support their goals. A second personal conference addresses Program Evaluation and individual project issues. The 
final week of the course has students submit revised personal ID models and a written self-evaluation of the course 
and their learning. 

Developmental Research Methodology 

To study the reflexive teaching approach, the design and development cycle (Richey & Nelson, 1996) was 
adopted as a research framework for six deliveries of an ID course from 1994-1998. In developmental research, 
objectives, rather than questions, characterize the inquiry. In this study we were interested in how the reflexive 
approach developed over time, and our research objectives for each delivery of the course included the following: 

• Describe the design decisions . 

• Describe the implementation of the model, or what occurred during each delivery. 

• Describe the evaluation of the model, in terms of student learning and student perceptions of their 
learning and teaching. 

The case study, defined as each class delivery, was chosen as the unit of analysis to describe the course 
development (Yin, 1994). Case 1 was a 5 -week summer course with nine contact hours per week. Cases 2-5 were 
15-week semesters, which met for three hours per week. Case 6 involved K-12 teachers from a school 
district/university-sponsored master’s program, during a 15-week spring semester, which met off-campus for 3 
hours once per week. 

Participants included 113 students and two instructors in a master’s level instructional design course from a 
university’s instructional technology graduate education program. Of the 113 students, 73 had teaching experience. 
Educational levels of interest included 18 elementary school, 15 middle school, 26 high school, 6 with an overall K- 
12 interest, 29 college, and 19 training. The largest content area focus of the participants included science and 
technology (19), followed by language (17), computing (14), and special education (12). 

Eight different data sources were used (see Figure 2), providing us with a means of data triangulation in 
which different sources of information informed the three research objectives (Yin, 1994). Data sources for design 
decisions included working logs, e-mail, and syllabi. Working logs documented our thinking and involvement in the 
ID course, including class presentations, learning tasks, student guide content, and teaching model representations. 
A syllabus recorded major design decisions for each case, including course purpose, instructional materials, 
assessment, and course sequence. 
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Data sources for implementation of the reflexive approach included working logs, e-mail, draft ID projects, 
and personal conference notes. Whenever possible during class, student comments and observations of class 
activities and student/instructor performance were recorded in working logs. Outside of class working logs recorded 
our perceptions of what occurred in class, summaries of weekly submissions of student work, and notes from weekly 
instructor meetings. E-mail was a source of instructor dialogue on weekly shifts in instruction and student needs. 
Draft ID projects provided evidence on how students transferred their ID process understanding into design 
decisions. The first of two personal conferences between instructors and student met early in the semester to discuss 
a student’s project choice and needs assessment strategy. 

Data sources for evaluation of the model included students' completed projects, course evaluations, student 
evaluations of their learning and written notes from a second personal conference. Required project components 
included a Mission Statement of beliefs about leaming/leamers/instruction. Project Intent Statement, Needs 
Assessment, Sequence, Assessment Plan, Instructional Framework, Instructional Media/Technology Plan, a 
Prototype Lesson, and a Program Evaluation Plan (formative and summative). Course evaluations included Likert- 
scale questions to record student perceptions of instruction, instructors and materials. Students also completed open- 
ended questions, which asked them to rate and/or comment on learning tasks and instructional materials. The self- 
evaluation task varied over each course delivery but typically asked students to summarize what they learned in the 
course, what they would like to learn next, and comment on their experiences with the course. Finally, a second 
student-teacher conference was conducted at the end of the course to address student questions on their projects. 

For each case the design decisions were reported by describing (a) participants, (b) learning tasks, (c) 
course sequence, (d) assessment, and (e) instructional materials. Analysis of the implementation of the model for 
each case was reported by describing student performance and responses to instruction and instructor’s assistance 
during (a) ID context activities, (b) ID process instruction, and (c) draft ID projects. Evaluation of the teaching 
model was reported on the basis of summarizing (a) student performance on the ID project, (b) students’ self- 
perceptions of their learning, and (c) instructor responsivity to student needs. Details of the data collection and 
analysis procedures can found in Shambaugh and Magliaro (2001). 

Developmental Study Conclusions 
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What we learned from this developmental study fall into two categories that also match our view of 
developmental research: a greater understanding of (a) a process; in this case, a reflexive teaching approach for ID 
instruction, and (b) the use of developmental research to study ID instruction. 

Reflexive model understanding. A summary of what we learned about the teaching model is elaborated 

below. 

• Articulate the theoretical basis for a reflexive approach : learning is constructed by the individual 
(Bruner, 1990), that there is a developmental interplay between one’s thinking and the social world 
(Brown, Collins & Duguid, 1989; Fosnot, 1996; Lave & Wenger, 1991), and that teaching 
supports learning construction through multiple forms of assistance (Tharp & Gallimore, 1988). 
Teaching is not viewed as content delivery or communicating knowledge but the development of 
learning environments to support individual knowledge construction (Duffy & Cunningham, 

1996). 

• The social system in which our reflexive teaching was embedded: a reflexive stance regards 
teacher and student as mutual learners with different roles, both involved in a critical, self- 
appraisal of their activity within the learning environments (i.e., the participation structures). 

• Classroom syntax: Out of this study the classroom participation structure was characterized by 
three phases, including (a) setting the stage, (b) representing understanding by instructor and 
student, and (c) debriefing. Our assistance to student learning was also depicted in a responsivity 
cycle involving (a) design and implementation; (b) mutual engagement, performance, and 
reflection; and (c) submit learning tasks, query instructors on issues, and respond to student 
concerns. 

• Principles of student reaction : How students reacted to the teaching are summarized by individual 
ID phases. 

• Support system requirements: In this model students were new to learning beliefs examination, 
were sometimes uncomfortable with submitting work- in -progress versus finished work, were new 
to detailed feedback from instructors, and required some time to take responsibility for design 
decisions responsive to learners versus what instructors wanted. 

• Requirements for co-participation : Joint student-instructor learning required a (a) willingness to 
share control and responsibility for learning, (b) a readiness for dialogic education, and (c) a 
genuine desire to be reflexive in one's teaching and learning. 

Developmental research understanding. Our understanding of what it meant to study our teaching using a 
development research framework included the following. 

• Model representation : our description of the model provides a basis for subsequent development 
of the model for purposes of exploration, prediction, and planning. 

• Model development: a theoretical basis for a model must be articulated and one must be clear as to 
the purposes of the developmental study to study the model; in particular, what knowledge, 
understanding, or predicting is being developed. 

• Data management: maintaining systematic data management procedures are crucial to track the 
evolution of the model prototype and to make any generalizations across time. 

• Individual and collaborative requirements: a reflexive disposition is needed to study one's 
teaching. 

Developmental Research Issues 

This section provides more details on methodology issues involved in the above developmental research 

study. 

Data Management and Analysis 

Data management in developmental research involves the procedures for a systematic, coherent process of 
data collection, storage and retrieval for the purpose of high quality, accessible data, the documentation of analysis, 
and retention of data (Huberman & Miles, 1998). In this study data were analyzed using the qualitative techniques 
of Miles and Huberman (1994), which consisted of data reduction from original data sources using categorical 
analysis (Spradley, 1980) and display of this reduced data in “frames” or tables that enabled conclusions to be 
drawn. The data analysis sequence included collecting data, reducing the data into frames, and reporting the 
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reduction in an analysis section. The data reduction documents for each case were kept in 3 -ring notebooks and each 
notebook was divided by data sources, a strategy that served to separate the data from the report and provided a 
means to organize the data and track the analysis sequence from data source to data reduction to data reporting. 

Describing design decisions was straightforward. Syllabi concisely recorded these as well as what was 
written in working logs, which recorded dialogue between instructors over time as to what ID content means and 
how to teach this content. Implementation of these design decisions in an actual course was also relatively clear to 
record in Working Logs in terms of what we as instructors implemented and how students reacted to these decisions 
from what they said and what they designed during the semester. Weekly submissions of draft ID project 
components were evaluated in terms of performance criteria. Criteria for each project component were 
communicated to students using a task sheet. 

Evaluation of student learning was more complicated. The ID projects were the principal source to indicate 
student performance of ID understanding and were analyzed for (a) completeness (i.e.. Were all of the project 
components in place?), (b) consistency of learning beliefs across design components, and (c) coherence of design 
components. For consistency, important ideas (e.g., assisting learners, skill proficiency, working together, problem 
solving, multiple instructional approaches) were identified from a Project Intent Statement. The ID project was 
reviewed to note whether or not these ideas were explicitly addressed in the projects. For example, if a student wrote 
about the importance of students working together, we looked for this feature in the instructional approach, 
activities, or prototype lesson. A Mission Statement recorded what students believed were important teaching and 
learning principles. The ID project was reviewed to note whether or not the features of the Mission Statement were 
addressed in the projects. For coherence, notations were recorded in terms of how each design component was 
represented. A judgment was made whether or not these design components appropriately supported each other, 
such as a match between assessment methods and teaching. 

Each case was a description of “what happened and how the course proceeded” using design decisions, 
model implementation, and evaluation of the model as a way to describe the use and results of the model (between- 
case analysis). Data displays, structured summaries, and tables allowed a condensed view of the data sources and 
revealed that some further analysis was needed, such as coding of structured summaries to reveal themes as well as 
to identify exceptions and differences. 

The summary of an across-case analysis of the six cases (across -case analysis) reported the changes in 
design decisions, implementation, and evaluation of the model (see Shambaugh & Magliaro, 2001). As Huberman 
and Miles (1998) have commented, “each case has a specific history -which we discard at our peril-but it is a history 
contained within the general principles that influence its development” (p. 194). This summary attempted to 
preserve the uniqueness of each case, yet also make comparisons along the developmental cycle based on repeat 
deliveries of the course. In an effort to extend external validity, what participants’ “did, said, or designed,” were 
examined in multiple settings. The description of the reflexive model, based on what was found fromthis analysis, 
provided a set of generalizations on how the model was implemented, as well as conditions necessary for its use. 

The danger to this generalization was that “multiple cases will be analyzed at high levels of inference, aggregating 
out the local webs of causality and ending with a smoothed set of generalizations that may not apply to any single 
case” (Huberman & Miles, p. 194). We did not average, for example, course evaluation results (i.e., from Likert 
scales) to avoid misinterpretation and superficiality and to preserve case uniqueness. The goal was to better 
understand the overall processes at work across the cases, including teacher and student thinking, participation, and 
teacher responsivity. 

In traditional instances of qualitative data collection and analysis, the research “shifts between cycles of 
inductive data collection and analysis to deductive cycles of testing and verification” (Huberman & Miles, 1998, p. 
198). In this study, sources of data were already in place prior to conceptualizing a study framework. However, the 
details of the framework and the subsequent data analysis of the six cases cycled back and forth to realize more 
appropriate matches of methodology and method to existing data sources and research objectives. The analytic cycle 
for this study could be better described as one that moved between conceptual framework, case analysis, and study 
purpose. Although being clear as to the purpose of a study is preferable before constructing a methodology, such 
clarity is not always possible due to the complexity of processes to be studied, data, and personal involvement over 
time. This reality requires teachers -researchers who feel comfortable about this dynamic movement and emergence 
of understanding. 

One possible source for bias in this study is the large amount of data, which may have led to missing 
important information or overweighing some findings due to focusing on a particular and large set of data. Personal 
involvement with the course also increased the possibility that recorded observations in working logs highlighted 
particular incidents while ignoring others. The working logs, however, served as a “re flexivity journal” (Carney, 
1990) and recorded observations or design decisions that would have been lost to our collective memories over the 
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five years of involvement. Personal involvement as co-instructors also implied a danger in being selective and 
overconfident with some data. Another shortcoming was not checking descriptions with each case of students and 
additional peer review outside of the co-instructor. 

To address these shortcomings, we used multiple data sources for triangulation to achieve an agreement of 
one data source with another. Multiple sources of data, such as working logs, e-mail, and syllabi, also provided 
different strengths and complemented each other. Syllabi, for example, compactly recorded design decisions, while 
working logs and email documented our thinking that influenced these decisions. The data sources were a mix of 
student-generated (i.e., conferences, ID projects, course and self-evaluations) and instructor-generated (i.e., working 
logs, e-mail, syllabi) data. 

During the analysis of these data sources we looked for contrasts, comparisons, and exemplars and reported 
these during the data reduction so as not to filter out outliers and extreme instances. Replication of the conceptual 
framework across multiple cases helped to provide evidence that what was described in each case was based upon 
the details of the instructional approach and uniqueness of the setting and participants. We were conscious to remain 
“descriptive” in the writing during the analysis of each case. 

Another means of addressing verification of methodology, findings, and conclusions was an “auditing” by 
the co-instructor. Through periodic reviews of methodology and analysis, inconsistencies in design decisions were 
identified and prompted for clarification. Such feedback characterized another aspect of our reflexive stance, the 
need to assume regular, ongoing, and self-conscious documentation of teaching. 

Balancing Design-Teaching and Developmental Research 

The design decisions, implementation, and evaluation of the ID course were event-driven, meaning that 
they served our instructional needs to watch, ask, and examine (Wolcott, 1992). These observations, interviews, and 
documents were in place prior to the conceptual framework of the study. As a result, the data sources were not as 
complete, tightly defined, or structured across the six cases if they had been researcher-driven. Some data sources, 
such as syllabi, course evaluations, and self-evaluations, evolved to suit the learning needs of the students. However, 
because we had presented preliminary findings at research conferences (e.g., Shambaugh & Magliaro, 1995, 1996), 
we had collected and stored data for each case, as well as conducted analysis with most of the data sources, although 
using different methodologies. These research efforts can be regarded as interim analyses in which we became 
familiar with studying teaching and learning products, developing procedures in recording observations and personal 
conferences, as well as retaining and analyzing documents. Over the six cases, we came to better understand the 
instructional setting, being sensitive to research opportunities and becoming more systematic in our data collection 
and management efforts, but also retaining instruction and responsiveness to learners as our top priority. 

Developmental Research Guidelines 

From this study we have become more aware of how desjgning/teaching can be informed by adherence to 
research tenets, as well as more trustworthy results that can be obtained by structuring teaching decisions and 
learning artifacts as data sources. The following guidelines for conducting developmental research are organized 
along the developmental research cycle, consisting of design, implementation, and evaluation components (see 
Figure 3) and are listed first by teaching decisions and in the second column by developmental research prompts. 

The guidelines address both designing and teaching, activities customarily viewed as separate, but are viewed by us 
as complementary. Design requires the involvement of practitioners who bring insight into practical implementation 
problems, and our belief that designer and teacher can be one and the same. 

Design Guidelines 

A major teaching decision is determining the purpose of the instructional intervention, through the use of 
course goals. A complementary developmental research prompt is being clear as to the purpose of the research. By 
answering the question, "What is this study about?" an appropriate methodology to study the research question(s) 
can be formulated. Developmental research objectives help to understand the complex phenomena at play in any 
educational intervention, so as to provide initial descriptive data for subsequent research questions. 

Design and teacher thinking can be made explicit by recording design conversations and decisions from 
notes, lesson plans, unit plans, curriculum guides, or syllabi. A more comprehensive representation of one's teaching 
can be documented in an instructional framework, using the conceptual approach of Joyce, Weil, and Calhoun 
(2000). The instructional framework records the theoretical learning foundation of the different teaching approaches. 
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describes the social and support systems of the approaches, specifies syntax or procedural guidelines of to 
implement the approaches, and the instructional and nurturant effects of the teaching. 

Another set of design/teaching decisions is to think through the activity roles of the teacher and the student 
and to identify the purpose of the activities, determine if the activities are developmentally appropriate, and 
sequenced appropriately. Thus, one can examine the alignment between desired learning outcomes, teaching 
options, and assessment. Developmental research prompts here would retain copies of learning activities and tasks 
so as to provide evidence of how these data sources evolved over time to support teaching adjustments. 

Implementation Guidelines 

Implementation guidelines for teachers involve ongoing evaluation of one's design/teaching decisions and 
dynamic adjustments needed during the intervention to address the desired learning outcomes. From a 
developmental research point of view, these design and/or teaching adjustments need to be recorded so as to provide 
a clear description of what occurred. Also, any changes in activities or tasks need to be documented to provide a 
description of how these data sources evolve over time. 

It is possible during implementation that other forms of learning may be occurring. A research prompt can 
be useful to continue looking for phenomena that may be occurring but not addressed through a methodology or that 
emerges as a result of what occurs from the intervention or from "looking at classrooms" in new methodological 
approaches. Each school year or course delivery provides a unique set of learners and learning characteristics that 
must be analyzed in developmental research as a unique case. 

During implementation teachers remain conscious of student perceptions to their teaching and the tasks and 
activities provided for them, as well as ongoing assessment of the ways in which students are learning or not 
learning. Are the assessment methods providing this information? Developmental research prompts here include 
habitual analysis of ongoing assessment, whether observations, interviews, or learning artifacts. In addition, the 
development of instruments to periodically assess student perceptions provides a new data source for 
implementation analysis and teaching adjustments. 

Evaluation Guidelines 

Evaluation guidelines primarily involve the examination of student performance on learning tasks and 
feedback on teaching efforts. Developmental research requires data sources that reveal student learning in terms of 
learning outcomes, as well as student or peer perceptions on teaching. Multiple forms of data from student activities 
(process and product forms) provide triangulating evidence of student learning. A challenge here is to maintain 
systematic data collection and management procedures in light of busy school schedules. 

A secondary set of evaluation guidelines encourages designers/teachers to solicit feedback and advice from 
peers and other sources. Developmental research prompts teachers to collaborate on teacher studies and to 
disseminate findings. 



Designing/Teaching Decisions 


Developmental Research Prompts 


DESIGN 


What is to be learned in this intervention? 


What is the purpose of the study? Be explicit about how 
teaching will be studied through research objectives 


Make explicit teacher thinking. Determine if there is an 
alignment between learning outcomes, teaching strategy, 
and assessment. 


Record and store design decisions from conversations, 
notes, or lesson plans, syllabi. 

Document instructional framework, including theoretical 
foundation of teaching (Joyce, Weil & Calhoun, 2000) 


Identify and describe teacher and student activity. 


Examine teacher and student activity as data through 
reflective and performance artifacts. Retain copies of 
learning tasks. 


IMPLEMENTATION 


Continually evaluate appropriateness of design decisions 
to learning outcomes. Document teaching activities and 
adjustments. 


Document changes in data sources as they evolve over 
time to address research objectives. 


What other forms of learning are occurring or not 
occurring? (incidental learning) 


Document direct (learning goals) and indirect (nurturant) 
effects of instruction (Joyce, Weil & Calhoun, 2000). 
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Determine learning characteristics of students. 


Each course delivery analyzed as a unique case. 


Examine assessment criteria. 


Are assessment criteria analyzable? 


How are students learning and perceiving your 
instruction? 


Develop instruments to obtain student learning and 
student perceptions. 


EVALUATION 


Keep records on student performance. 


Identify data sources that reveal student learning. 
Maintain systematic data management procedures. 


Solicit feedback from students and listen to what they 
say. 


Identify data sources for student perceptions of teaching. 
Maintain systematic data management procedures. 


Seek out feedback from peers, develop teaching 
repertoire. 


Collaborate on research into one’s teaching, disseminate 
findings. 



Fi 2 ure 3 . Designing/teaching and developmental research guidelines. 



Implications 

The downsides of research for designers -teachers have been characterized by van den Akker (1999) as 
answers that are frequently too narrow, too superficial, or too late to do any good. Developmental research aims to 
address these issues of meaningfulness, generalizability, and usefulness. Developmental research can be used to 
assist designers and teachers in the development of educational interventions, while providing a systematic means to 
study their implementation. As all interventions are influenced by multiple stakeholders and are contextually -rich in 
nature, developmental research provides a dynamic vantage point for collaborators to talk about their roles, whether 
these roles be pragmatic (design, teaching) or knowledge-building (research). Developmental research also provides 
a framework to study instructional problems and responses, particularly as changes in the methodological 
framework may be required to adapt to evolving response prototypes. 

Reeves (2000) provides several heuristics for developmental research activities, including the need to focus 
on difficult learning problems; align designs with learning outcomes, teaching, and assessment; collaborate and 
share with others; and the hard work needed for any developmental research project. Another critical heuristic 
suggested by Reeves involves identifying the theoretical and practical design principles that underlie a prototype and 
conduct rigorous studies of these principles in real settings. Carroll (2000) reminds us that viewing design as inquiry 
"raises the question of what abstractions can support the development and sense of knowledge in design" (p. 65). In 
particular, we raise to ourselves and others two challenges; namely, (a) the difficulty of abstracting principles or 
frameworks to help us in designing contextually -rich learning environments, and (b) acknowledging in 
developmental research the naturalistic nature of teacher knowledge. Both of these challenges resonate with the twin 
purposes of developmental research raised at the beginning of this paper: (a) formulating what we learn from our 
design efforts and (b) developing responsive educational interventions. 
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